Crowd Truth: Harnessing disagreement in crowdsourcing a relation extraction gold standard

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. WebSci’13, May 1 – May 5, 2013, Paris, France. Copyright 2013 ACM 978-1-4503-1889-1....$10.00. Abstract One of the first steps in designing NLP solutions, and indeed many web data analytics, is creating a human annotated ground truth, typically based on the assumption that for each annotated instance there is a single right answer. From this assumption it has always followed that ground truth quality can be measured in inter-annotator agreement. We challenge this assumption by observing that for certain annotation tasks, annotator disagreement reflects semantic ambiguity in the target instances and provides useful information. We propose a new type of ground truth, a crowd truth, which is richer in diversity of perspectives and interpretations, and reflects more realistic human knowledge. We propose a framework to exploit such diverse human responses to annotation tasks for analyzing and understanding disagreement. Our discussion centers on a use case of relation extraction from medical text, however the crowd truth idea generalizes to most crowdsourced annotation tasks.